Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Improve E2E test consistency #55554

Merged
merged 3 commits into from
Oct 24, 2023

Conversation

dcalhoun
Copy link
Member

What?

Improve E2E test environment consistency.

Why?

Cryptic errors from inconsistent environments are confusing.

How?

test: Mitigate conflicts between development server caches
Occasionally, cryptic npm package dependency errors can occur between
test runs. They appear to originate from stale development caches, e.g.
Metro server. Clearing these caches, which E2E tests should not rely
upon mitigates the likelihood of these errors.

test: Prefix informative E2E test setup logs
Merely an attempt to add structure to the outputted logs.

test: Avoid setup scripts during test runs
There may be clarity and efficiency by having an explicit setup script.
It was avoid running unnecessary scripts during test runs. It is also
referenced in the current documentation as a script to run
intentionally.

Testing Instructions

  • Verify the npm run native test:e2e:setup command succeeds when run.
  • Verify the npm run native test:e2e:ios:local command does not throw errors.
  • Verify the npm run native test:e2e:android:local command does not throw
    errors.

Testing Instructions for Keyboard

n/a

Screenshots or screencast

n/a

Occasionally, cryptic npm package dependency errors can occur between
test runs. They appear to originate from stale development caches, e.g.
Metro server. Clearing these caches, which E2E tests should not rely
upon mitigates the likelihood of these errors.
Merely an attempt to add structure to the outputted logs.
@dcalhoun dcalhoun added [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. Mobile App - i.e. Android or iOS Native mobile impl of the block editor. (Note: used in scripts, ping mobile folks to change) labels Oct 23, 2023
@github-actions
Copy link

github-actions bot commented Oct 23, 2023

Size Change: 0 B

Total Size: 1.69 MB

ℹ️ View Unchanged
Filename Size
build/a11y/index.min.js 964 B
build/annotations/index.min.js 2.71 kB
build/api-fetch/index.min.js 2.29 kB
build/autop/index.min.js 2.11 kB
build/blob/index.min.js 461 B
build/block-directory/index.min.js 7.25 kB
build/block-directory/style-rtl.css 1.04 kB
build/block-directory/style.css 1.04 kB
build/block-editor/content-rtl.css 4.28 kB
build/block-editor/content.css 4.27 kB
build/block-editor/default-editor-styles-rtl.css 403 B
build/block-editor/default-editor-styles.css 403 B
build/block-editor/index.min.js 247 kB
build/block-editor/style-rtl.css 15.6 kB
build/block-editor/style.css 15.6 kB
build/block-library/blocks/archives/editor-rtl.css 61 B
build/block-library/blocks/archives/editor.css 60 B
build/block-library/blocks/archives/style-rtl.css 90 B
build/block-library/blocks/archives/style.css 90 B
build/block-library/blocks/audio/editor-rtl.css 150 B
build/block-library/blocks/audio/editor.css 150 B
build/block-library/blocks/audio/style-rtl.css 122 B
build/block-library/blocks/audio/style.css 122 B
build/block-library/blocks/audio/theme-rtl.css 138 B
build/block-library/blocks/audio/theme.css 138 B
build/block-library/blocks/avatar/editor-rtl.css 116 B
build/block-library/blocks/avatar/editor.css 116 B
build/block-library/blocks/avatar/style-rtl.css 104 B
build/block-library/blocks/avatar/style.css 104 B
build/block-library/blocks/block/editor-rtl.css 305 B
build/block-library/blocks/block/editor.css 305 B
build/block-library/blocks/button/editor-rtl.css 587 B
build/block-library/blocks/button/editor.css 587 B
build/block-library/blocks/button/style-rtl.css 633 B
build/block-library/blocks/button/style.css 632 B
build/block-library/blocks/buttons/editor-rtl.css 337 B
build/block-library/blocks/buttons/editor.css 337 B
build/block-library/blocks/buttons/style-rtl.css 332 B
build/block-library/blocks/buttons/style.css 332 B
build/block-library/blocks/calendar/style-rtl.css 239 B
build/block-library/blocks/calendar/style.css 239 B
build/block-library/blocks/categories/editor-rtl.css 113 B
build/block-library/blocks/categories/editor.css 112 B
build/block-library/blocks/categories/style-rtl.css 124 B
build/block-library/blocks/categories/style.css 124 B
build/block-library/blocks/code/editor-rtl.css 53 B
build/block-library/blocks/code/editor.css 53 B
build/block-library/blocks/code/style-rtl.css 121 B
build/block-library/blocks/code/style.css 121 B
build/block-library/blocks/code/theme-rtl.css 124 B
build/block-library/blocks/code/theme.css 124 B
build/block-library/blocks/columns/editor-rtl.css 108 B
build/block-library/blocks/columns/editor.css 108 B
build/block-library/blocks/columns/style-rtl.css 421 B
build/block-library/blocks/columns/style.css 421 B
build/block-library/blocks/comment-author-avatar/editor-rtl.css 125 B
build/block-library/blocks/comment-author-avatar/editor.css 125 B
build/block-library/blocks/comment-content/style-rtl.css 92 B
build/block-library/blocks/comment-content/style.css 92 B
build/block-library/blocks/comment-template/style-rtl.css 199 B
build/block-library/blocks/comment-template/style.css 198 B
build/block-library/blocks/comments-pagination-numbers/editor-rtl.css 123 B
build/block-library/blocks/comments-pagination-numbers/editor.css 121 B
build/block-library/blocks/comments-pagination/editor-rtl.css 222 B
build/block-library/blocks/comments-pagination/editor.css 209 B
build/block-library/blocks/comments-pagination/style-rtl.css 235 B
build/block-library/blocks/comments-pagination/style.css 231 B
build/block-library/blocks/comments-title/editor-rtl.css 75 B
build/block-library/blocks/comments-title/editor.css 75 B
build/block-library/blocks/comments/editor-rtl.css 840 B
build/block-library/blocks/comments/editor.css 839 B
build/block-library/blocks/comments/style-rtl.css 637 B
build/block-library/blocks/comments/style.css 636 B
build/block-library/blocks/cover/editor-rtl.css 647 B
build/block-library/blocks/cover/editor.css 650 B
build/block-library/blocks/cover/style-rtl.css 1.7 kB
build/block-library/blocks/cover/style.css 1.69 kB
build/block-library/blocks/details/editor-rtl.css 65 B
build/block-library/blocks/details/editor.css 65 B
build/block-library/blocks/details/style-rtl.css 98 B
build/block-library/blocks/details/style.css 98 B
build/block-library/blocks/embed/editor-rtl.css 293 B
build/block-library/blocks/embed/editor.css 293 B
build/block-library/blocks/embed/style-rtl.css 410 B
build/block-library/blocks/embed/style.css 410 B
build/block-library/blocks/embed/theme-rtl.css 138 B
build/block-library/blocks/embed/theme.css 138 B
build/block-library/blocks/file/editor-rtl.css 316 B
build/block-library/blocks/file/editor.css 316 B
build/block-library/blocks/file/style-rtl.css 311 B
build/block-library/blocks/file/style.css 312 B
build/block-library/blocks/file/view.min.js 321 B
build/block-library/blocks/footnotes/style-rtl.css 201 B
build/block-library/blocks/footnotes/style.css 199 B
build/block-library/blocks/form-input/editor-rtl.css 229 B
build/block-library/blocks/form-input/editor.css 228 B
build/block-library/blocks/form-input/style-rtl.css 343 B
build/block-library/blocks/form-input/style.css 343 B
build/block-library/blocks/form-submission-notification/editor-rtl.css 343 B
build/block-library/blocks/form-submission-notification/editor.css 342 B
build/block-library/blocks/form-submit-button/style-rtl.css 69 B
build/block-library/blocks/form-submit-button/style.css 69 B
build/block-library/blocks/form/view.min.js 452 B
build/block-library/blocks/freeform/editor-rtl.css 2.61 kB
build/block-library/blocks/freeform/editor.css 2.61 kB
build/block-library/blocks/gallery/editor-rtl.css 957 B
build/block-library/blocks/gallery/editor.css 962 B
build/block-library/blocks/gallery/style-rtl.css 1.55 kB
build/block-library/blocks/gallery/style.css 1.55 kB
build/block-library/blocks/gallery/theme-rtl.css 122 B
build/block-library/blocks/gallery/theme.css 122 B
build/block-library/blocks/group/editor-rtl.css 654 B
build/block-library/blocks/group/editor.css 654 B
build/block-library/blocks/group/style-rtl.css 57 B
build/block-library/blocks/group/style.css 57 B
build/block-library/blocks/group/theme-rtl.css 78 B
build/block-library/blocks/group/theme.css 78 B
build/block-library/blocks/heading/style-rtl.css 189 B
build/block-library/blocks/heading/style.css 189 B
build/block-library/blocks/html/editor-rtl.css 340 B
build/block-library/blocks/html/editor.css 341 B
build/block-library/blocks/image/editor-rtl.css 834 B
build/block-library/blocks/image/editor.css 833 B
build/block-library/blocks/image/style-rtl.css 1.54 kB
build/block-library/blocks/image/style.css 1.54 kB
build/block-library/blocks/image/theme-rtl.css 137 B
build/block-library/blocks/image/theme.css 137 B
build/block-library/blocks/image/view.min.js 2.01 kB
build/block-library/blocks/latest-comments/style-rtl.css 357 B
build/block-library/blocks/latest-comments/style.css 357 B
build/block-library/blocks/latest-posts/editor-rtl.css 213 B
build/block-library/blocks/latest-posts/editor.css 212 B
build/block-library/blocks/latest-posts/style-rtl.css 478 B
build/block-library/blocks/latest-posts/style.css 478 B
build/block-library/blocks/list/style-rtl.css 88 B
build/block-library/blocks/list/style.css 88 B
build/block-library/blocks/media-text/editor-rtl.css 266 B
build/block-library/blocks/media-text/editor.css 263 B
build/block-library/blocks/media-text/style-rtl.css 505 B
build/block-library/blocks/media-text/style.css 503 B
build/block-library/blocks/more/editor-rtl.css 431 B
build/block-library/blocks/more/editor.css 431 B
build/block-library/blocks/navigation-link/editor-rtl.css 671 B
build/block-library/blocks/navigation-link/editor.css 672 B
build/block-library/blocks/navigation-link/style-rtl.css 103 B
build/block-library/blocks/navigation-link/style.css 103 B
build/block-library/blocks/navigation-submenu/editor-rtl.css 299 B
build/block-library/blocks/navigation-submenu/editor.css 299 B
build/block-library/blocks/navigation/editor-rtl.css 2.26 kB
build/block-library/blocks/navigation/editor.css 2.26 kB
build/block-library/blocks/navigation/style-rtl.css 2.26 kB
build/block-library/blocks/navigation/style.css 2.25 kB
build/block-library/blocks/navigation/view.min.js 1.07 kB
build/block-library/blocks/nextpage/editor-rtl.css 395 B
build/block-library/blocks/nextpage/editor.css 395 B
build/block-library/blocks/page-list/editor-rtl.css 401 B
build/block-library/blocks/page-list/editor.css 401 B
build/block-library/blocks/page-list/style-rtl.css 175 B
build/block-library/blocks/page-list/style.css 175 B
build/block-library/blocks/paragraph/editor-rtl.css 235 B
build/block-library/blocks/paragraph/editor.css 235 B
build/block-library/blocks/paragraph/style-rtl.css 335 B
build/block-library/blocks/paragraph/style.css 335 B
build/block-library/blocks/post-author/style-rtl.css 175 B
build/block-library/blocks/post-author/style.css 176 B
build/block-library/blocks/post-comments-form/editor-rtl.css 96 B
build/block-library/blocks/post-comments-form/editor.css 96 B
build/block-library/blocks/post-comments-form/style-rtl.css 508 B
build/block-library/blocks/post-comments-form/style.css 508 B
build/block-library/blocks/post-date/style-rtl.css 61 B
build/block-library/blocks/post-date/style.css 61 B
build/block-library/blocks/post-excerpt/editor-rtl.css 71 B
build/block-library/blocks/post-excerpt/editor.css 71 B
build/block-library/blocks/post-excerpt/style-rtl.css 141 B
build/block-library/blocks/post-excerpt/style.css 141 B
build/block-library/blocks/post-featured-image/editor-rtl.css 588 B
build/block-library/blocks/post-featured-image/editor.css 586 B
build/block-library/blocks/post-featured-image/style-rtl.css 345 B
build/block-library/blocks/post-featured-image/style.css 345 B
build/block-library/blocks/post-navigation-link/style-rtl.css 215 B
build/block-library/blocks/post-navigation-link/style.css 214 B
build/block-library/blocks/post-template/editor-rtl.css 99 B
build/block-library/blocks/post-template/editor.css 98 B
build/block-library/blocks/post-template/style-rtl.css 409 B
build/block-library/blocks/post-template/style.css 408 B
build/block-library/blocks/post-terms/style-rtl.css 96 B
build/block-library/blocks/post-terms/style.css 96 B
build/block-library/blocks/post-time-to-read/style-rtl.css 69 B
build/block-library/blocks/post-time-to-read/style.css 69 B
build/block-library/blocks/post-title/style-rtl.css 100 B
build/block-library/blocks/post-title/style.css 100 B
build/block-library/blocks/preformatted/style-rtl.css 125 B
build/block-library/blocks/preformatted/style.css 125 B
build/block-library/blocks/pullquote/editor-rtl.css 135 B
build/block-library/blocks/pullquote/editor.css 135 B
build/block-library/blocks/pullquote/style-rtl.css 335 B
build/block-library/blocks/pullquote/style.css 335 B
build/block-library/blocks/pullquote/theme-rtl.css 168 B
build/block-library/blocks/pullquote/theme.css 168 B
build/block-library/blocks/query-pagination-numbers/editor-rtl.css 122 B
build/block-library/blocks/query-pagination-numbers/editor.css 121 B
build/block-library/blocks/query-pagination/editor-rtl.css 221 B
build/block-library/blocks/query-pagination/editor.css 211 B
build/block-library/blocks/query-pagination/style-rtl.css 288 B
build/block-library/blocks/query-pagination/style.css 284 B
build/block-library/blocks/query-title/style-rtl.css 63 B
build/block-library/blocks/query-title/style.css 63 B
build/block-library/blocks/query/editor-rtl.css 486 B
build/block-library/blocks/query/editor.css 486 B
build/block-library/blocks/query/style-rtl.css 312 B
build/block-library/blocks/query/style.css 308 B
build/block-library/blocks/query/view.min.js 609 B
build/block-library/blocks/quote/style-rtl.css 222 B
build/block-library/blocks/quote/style.css 222 B
build/block-library/blocks/quote/theme-rtl.css 223 B
build/block-library/blocks/quote/theme.css 226 B
build/block-library/blocks/read-more/style-rtl.css 132 B
build/block-library/blocks/read-more/style.css 132 B
build/block-library/blocks/rss/editor-rtl.css 149 B
build/block-library/blocks/rss/editor.css 149 B
build/block-library/blocks/rss/style-rtl.css 289 B
build/block-library/blocks/rss/style.css 288 B
build/block-library/blocks/search/editor-rtl.css 184 B
build/block-library/blocks/search/editor.css 184 B
build/block-library/blocks/search/style-rtl.css 613 B
build/block-library/blocks/search/style.css 613 B
build/block-library/blocks/search/theme-rtl.css 114 B
build/block-library/blocks/search/theme.css 114 B
build/block-library/blocks/search/view.min.js 471 B
build/block-library/blocks/separator/editor-rtl.css 146 B
build/block-library/blocks/separator/editor.css 146 B
build/block-library/blocks/separator/style-rtl.css 234 B
build/block-library/blocks/separator/style.css 234 B
build/block-library/blocks/separator/theme-rtl.css 194 B
build/block-library/blocks/separator/theme.css 194 B
build/block-library/blocks/shortcode/editor-rtl.css 329 B
build/block-library/blocks/shortcode/editor.css 329 B
build/block-library/blocks/site-logo/editor-rtl.css 760 B
build/block-library/blocks/site-logo/editor.css 760 B
build/block-library/blocks/site-logo/style-rtl.css 204 B
build/block-library/blocks/site-logo/style.css 204 B
build/block-library/blocks/site-tagline/editor-rtl.css 86 B
build/block-library/blocks/site-tagline/editor.css 86 B
build/block-library/blocks/site-title/editor-rtl.css 116 B
build/block-library/blocks/site-title/editor.css 116 B
build/block-library/blocks/site-title/style-rtl.css 57 B
build/block-library/blocks/site-title/style.css 57 B
build/block-library/blocks/social-link/editor-rtl.css 184 B
build/block-library/blocks/social-link/editor.css 184 B
build/block-library/blocks/social-links/editor-rtl.css 682 B
build/block-library/blocks/social-links/editor.css 681 B
build/block-library/blocks/social-links/style-rtl.css 1.45 kB
build/block-library/blocks/social-links/style.css 1.45 kB
build/block-library/blocks/spacer/editor-rtl.css 359 B
build/block-library/blocks/spacer/editor.css 359 B
build/block-library/blocks/spacer/style-rtl.css 48 B
build/block-library/blocks/spacer/style.css 48 B
build/block-library/blocks/table/editor-rtl.css 432 B
build/block-library/blocks/table/editor.css 432 B
build/block-library/blocks/table/style-rtl.css 646 B
build/block-library/blocks/table/style.css 645 B
build/block-library/blocks/table/theme-rtl.css 157 B
build/block-library/blocks/table/theme.css 157 B
build/block-library/blocks/tag-cloud/style-rtl.css 251 B
build/block-library/blocks/tag-cloud/style.css 253 B
build/block-library/blocks/template-part/editor-rtl.css 403 B
build/block-library/blocks/template-part/editor.css 403 B
build/block-library/blocks/template-part/theme-rtl.css 101 B
build/block-library/blocks/template-part/theme.css 101 B
build/block-library/blocks/term-description/style-rtl.css 111 B
build/block-library/blocks/term-description/style.css 111 B
build/block-library/blocks/text-columns/editor-rtl.css 95 B
build/block-library/blocks/text-columns/editor.css 95 B
build/block-library/blocks/text-columns/style-rtl.css 166 B
build/block-library/blocks/text-columns/style.css 166 B
build/block-library/blocks/verse/style-rtl.css 99 B
build/block-library/blocks/verse/style.css 99 B
build/block-library/blocks/video/editor-rtl.css 552 B
build/block-library/blocks/video/editor.css 555 B
build/block-library/blocks/video/style-rtl.css 191 B
build/block-library/blocks/video/style.css 191 B
build/block-library/blocks/video/theme-rtl.css 139 B
build/block-library/blocks/video/theme.css 139 B
build/block-library/classic-rtl.css 179 B
build/block-library/classic.css 179 B
build/block-library/common-rtl.css 1.11 kB
build/block-library/common.css 1.11 kB
build/block-library/editor-elements-rtl.css 75 B
build/block-library/editor-elements.css 75 B
build/block-library/editor-rtl.css 12.4 kB
build/block-library/editor.css 12.4 kB
build/block-library/elements-rtl.css 54 B
build/block-library/elements.css 54 B
build/block-library/index.min.js 211 kB
build/block-library/reset-rtl.css 472 B
build/block-library/reset.css 472 B
build/block-library/style-rtl.css 14.4 kB
build/block-library/style.css 14.4 kB
build/block-library/theme-rtl.css 700 B
build/block-library/theme.css 705 B
build/block-serialization-default-parser/index.min.js 1.13 kB
build/block-serialization-spec-parser/index.min.js 2.87 kB
build/blocks/index.min.js 51.5 kB
build/commands/index.min.js 15.5 kB
build/commands/style-rtl.css 947 B
build/commands/style.css 942 B
build/components/index.min.js 251 kB
build/components/style-rtl.css 11.9 kB
build/components/style.css 11.9 kB
build/compose/index.min.js 12.7 kB
build/core-commands/index.min.js 2.72 kB
build/core-data/index.min.js 71.7 kB
build/customize-widgets/index.min.js 12 kB
build/customize-widgets/style-rtl.css 1.51 kB
build/customize-widgets/style.css 1.5 kB
build/data-controls/index.min.js 651 B
build/data/index.min.js 8.87 kB
build/date/index.min.js 17.9 kB
build/deprecated/index.min.js 462 B
build/dom-ready/index.min.js 336 B
build/dom/index.min.js 4.68 kB
build/edit-post/classic-rtl.css 571 B
build/edit-post/classic.css 571 B
build/edit-post/index.min.js 35.7 kB
build/edit-post/style-rtl.css 7.88 kB
build/edit-post/style.css 7.88 kB
build/edit-site/index.min.js 206 kB
build/edit-site/style-rtl.css 14.3 kB
build/edit-site/style.css 14.3 kB
build/edit-widgets/index.min.js 17 kB
build/edit-widgets/style-rtl.css 4.85 kB
build/edit-widgets/style.css 4.84 kB
build/editor/index.min.js 46.2 kB
build/editor/style-rtl.css 3.58 kB
build/editor/style.css 3.58 kB
build/element/index.min.js 4.87 kB
build/escape-html/index.min.js 548 B
build/format-library/index.min.js 7.82 kB
build/format-library/style-rtl.css 577 B
build/format-library/style.css 577 B
build/hooks/index.min.js 1.57 kB
build/html-entities/index.min.js 454 B
build/i18n/index.min.js 3.61 kB
build/interactivity/index.min.js 11.4 kB
build/is-shallow-equal/index.min.js 535 B
build/keyboard-shortcuts/index.min.js 1.76 kB
build/keycodes/index.min.js 1.9 kB
build/list-reusable-blocks/index.min.js 2.21 kB
build/list-reusable-blocks/style-rtl.css 865 B
build/list-reusable-blocks/style.css 865 B
build/media-utils/index.min.js 2.92 kB
build/notices/index.min.js 964 B
build/nux/index.min.js 2.01 kB
build/nux/style-rtl.css 775 B
build/nux/style.css 771 B
build/patterns/index.min.js 4.81 kB
build/patterns/style-rtl.css 657 B
build/patterns/style.css 657 B
build/plugins/index.min.js 1.81 kB
build/preferences-persistence/index.min.js 1.85 kB
build/preferences/index.min.js 1.26 kB
build/primitives/index.min.js 994 B
build/priority-queue/index.min.js 1.52 kB
build/private-apis/index.min.js 988 B
build/react-i18n/index.min.js 631 B
build/react-refresh-entry/index.min.js 9.46 kB
build/react-refresh-runtime/index.min.js 6.78 kB
build/redux-routine/index.min.js 2.71 kB
build/reusable-blocks/index.min.js 2.73 kB
build/reusable-blocks/style-rtl.css 265 B
build/reusable-blocks/style.css 265 B
build/rich-text/index.min.js 10.2 kB
build/router/index.min.js 1.79 kB
build/server-side-render/index.min.js 1.96 kB
build/shortcode/index.min.js 1.4 kB
build/style-engine/index.min.js 1.98 kB
build/token-list/index.min.js 587 B
build/url/index.min.js 3.83 kB
build/vendors/inert-polyfill.min.js 2.48 kB
build/vendors/react-dom.min.js 41.8 kB
build/vendors/react.min.js 4.02 kB
build/viewport/index.min.js 967 B
build/warning/index.min.js 259 B
build/widgets/index.min.js 7.18 kB
build/widgets/style-rtl.css 1.18 kB
build/widgets/style.css 1.18 kB
build/wordcount/index.min.js 1.03 kB

compressed-size-action

@@ -112,11 +112,11 @@
"test:e2e:setup": "./bin/test-e2e-setup.sh",
"test:e2e:bundle:android": "mkdir -p android/app/src/main/assets && npm run rn-bundle -- --reset-cache --platform android --dev false --minify false --entry-file index.js --bundle-output android/app/src/main/assets/index.android.bundle --assets-dest android/app/src/main/res",
"test:e2e:build-app:android": "cd android && ./gradlew clean && ./gradlew assembleDebug",
"test:e2e:android:local": "npm run test:e2e:setup && npm run test:e2e:bundle:android && npm run test:e2e:build-app:android && TEST_RN_PLATFORM=android npm run device-tests:local",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change treats test:e2e:setup as a one-off command run when setting up or repairing the test environment, rather than running the test:e2e:setup command on every test run. WDYT?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change makes sense to me 👍

@dcalhoun dcalhoun marked this pull request as ready for review October 23, 2023 20:10
@dcalhoun dcalhoun requested a review from geriux October 23, 2023 20:10
@github-actions
Copy link

github-actions bot commented Oct 23, 2023

Flaky tests detected in 0f6ad7d.
Some tests passed with failed attempts. The failures may not be related to this commit but are still reported for visibility. See the documentation for more information.

🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/6620335365
📝 Reported issues:

There may be clarity and efficiency by having an explicit setup script.
It was avoid running unnecessary scripts during test runs. It is also
referenced in the current documentation as a script to run
intentionally.
@dcalhoun dcalhoun force-pushed the test/improve-e2e-test-consistency branch from 4f33f97 to 0f6ad7d Compare October 23, 2023 23:47
Copy link
Member

@geriux geriux left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 🚀

@dcalhoun dcalhoun merged commit 9d4613a into rnmobile/appium-2 Oct 24, 2023
50 checks passed
@dcalhoun dcalhoun deleted the test/improve-e2e-test-consistency branch October 24, 2023 13:49
dcalhoun added a commit that referenced this pull request Oct 24, 2023
* Dependencies update:
- Bumps Appium to version 2.0.0
- Removes wd in favor of WebdriverIO

* - Adds appium:setup script to install needed drivers for Appium.
- Updates test:e2e:build-wda with the new Appium home path for the xcuitest driver.
- Updates test:e2e:android:local and test:e2e:ios:local to add the newly added appium:setup script to install drivers if needed.

* Update Android Runner to cache the Appium drivers

* Update iOS Runner - Bump Xcode to 14.2

* Update iOS Runner with these changes:

- Move the Launch of the simulator step to the top so it starts booting up the simulator in the background, so it doesn't block the next steps while it's stating.
- Install and cache the Appium drivers.

* Updates the E2E capabilities settings:

- It removes unneeded values after the migration.
- Bumps the iOS platform version to 16.2.
- Adds specific SauceLabs configuration that sets the Appium version to 2.0.0.
- Adds a prefixKeysWithAppium util to append the "appium:" prefix for the Appium capabilities settings. This is added to simplify the modifications we do for some of the settings in runtime.

* Updates the E2E serverConfigs file with the updated SauceLabs settings for Appium 2 and WebdriverIO

* E2E Utils:

- Remove import of wd and set it as null temporarily while we migrate its usage to WebdriverIO

* E2E Utils:

Update getIOSPlatformVersions to get the desired platform version from the capabilities settings instead of an inline value

* E2E Utils:

- Update Remote configuration with the new structure for WebdriverIO and Appium 2
- Uses the prefixKeysWithAppium util for the Appium capabilities, we cannot use "appium:options" since that's only available for real devices and we use simulators instead.
- The app capability value was updated to the new storage:filename= prefix this was important to make SauceLabs to work correctly

* E2E Utils - Update the stopDriver function with:

- Changes sessionID to sessionId
- Removes the hash value to append in the Job URL, these URLs are public so it's not needed to append the auth param.
- Updates driver.quit() to driver.deleteSession()

* E2E Utils - Updates swipeFromTo to WebdriverIO

* E2E Utils - Updates isEditorVisible to WebdriverIO

It adds a custom timeout to waitForDisplayed, the simulators in CI are slow so we need to give enough time to find the post title locator.

* Remove usage of sauceJobStatus, WebdriverIO internally handles this

* EditorPage - Remove import of wd and keep a null value while we migrate to WebdriverIO

* EditorPage - Migrate initializeEditor to WebdriverIO

* Gutenberg Editor Rendering Media Blocks test - Migrate to WebdriverIO

* E2E Utils - Revert removing passing the auth token for the jobURL, it also updates the logic to the latest implementation from SauceLabs docs.

* Remove idleTimeout desktop-only SauceLabs option

* [Mobile] - Update E2E Block Insertion tests (#55167)

* E2E Utils, updates the typing helpers:

- Removes the platform specfic typing functionality in favor of WebdriverIO's addValue

* E2E Utils - Updates getKeycode and pressKeycode to WebdriverIO

* E2E Utils - Update tapStatusBariOS to WebdriverIO

* E2E Utils - Update toggleHtmlMode to WebdriverIO

* E2E Utils - Update waitForVisible to WebdriverIO

* E2E Editor Page - Update getBlockAtPosition to WebdriverIO

* E2E Editor Page - Update getTitleElement to WebdriverIO

* E2E Editor Page - Update getHtmlContent to WebdriverIO

* E2E Editor Page - Update swipeToolbarToElement to WebdriverIO

* E2E Editor Page - Update addNewBlock to WebdriverIO

* E2E Editor Page - Update findBlockButton to WebdriverIO

* E2E Editor Page - Update assertSlashInserterPresent to WebdriverIO

* E2E EditorPage - Update waitForElementToBeDisplayedById to WebdriverIO

* E2E Test - Block Insertion - Update to WebdriverIO

* Update typeString scroll delay comment

Co-authored-by: David Calhoun <github@davidcalhoun.me>

---------

Co-authored-by: David Calhoun <github@davidcalhoun.me>

* test: Update Search tests for Appium 2 (#55170)

* test: Update getSearchBlockTextElement query

Update syntax for WebdriverIO, which is a part of the Appium 2 upgrade.

* test: Update waitForElementToBeDisplayedByXPath query

Update syntax for WebdriverIO, which is a part of the Appium 2 upgrade.

* test: Update verifySearchElementText query

Update syntax for WebdriverIO, which is a part of the Appium 2 upgrade.

* test: Update getSearchBlockTextElement query

Update syntax for WebdriverIO, which is a part of the Appium 2 upgrade.

* E2E Editor Page - Update dismissKeyboard to use WebdiverIO's hideKeyboard (#55211)

* test: Update Media Blocks tests for Appium 2 (#55222)

* test: Replace sleep with pause utility

Update syntax for WebdriverIO, which is a part of the Appium 2 upgrade.

* test: Replace elementsByAccessibilityId usage

Update syntax for WebdriverIO, which is a part of the Appium 2 upgrade.

* test: Replace elementByXPath usage

Update syntax for WebdriverIO, which is a part of the Appium 2 upgrade.

* test: Repair enterCaptionToSelectedImageBlock utility

Update syntax for WebdriverIO, which is a part of the Appium 2 upgrade.

Additionally, the `clearValue` method did not succeed on the caption
Button element. Hence, the utility was expanded to instead type into the
TextView.

* test: Update Paragraph tests for Appium 2 (#55215)

* test: Update clickMiddleOfElement helper

Update syntax for WebdriverIO, which is a part of the Appium 2 upgrade.

* test: Update clickBeginningOfElement helper

Update syntax for WebdriverIO, which is a part of the Appium 2 upgrade.

* test: Update getTextForParagraphBlockAtPosition helper

Update syntax for WebdriverIO, which is a part of the Appium 2 upgrade.

* test: Replace elementsByXPath usage

Update syntax for WebdriverIO, which is a part of the Appium 2 upgrade.

* E2E Utils - Migrate dragAndDropAfterElement to WebdriverIO (#55254)

* [Mobile] - Move Appium drivers setup to bash script (#55257)

* Move Appium drivers setup to bash script

* Update drivers' names casing for consistency

* Rename appium:setup to test:e2e:setup

* test: Update E2E utilities (#55322)

* test: Set static device pixel ratio

WebdriverIO (the replacement for the deprecated WD driver library) does
not have a method for retrieving a device's pixel ratio. Therefore, this
sets an explicit value for each of our test devices.

* test: Update the `getAddBlockButton` utility

Update syntax for WebdriverIO, which is a part of the Appium 2 upgrade.

* test: Update toggleDarkMode utility

Seems this method has changed or been removed for Android.

* test: Increase robustness of waitForKeyboardToBeHidden utility

This utility appeared to fail occasionally due to an empty `addButton`
value.

* test: toggleDarkMode uses consistent approach across platforms

This updates the iOS variant to mirror the structure of the Android
variant in the name of consistency.

* refactor: Rename devicePixelRatio to pixelRatio

Mirror Android implementation found in Appium 2.

* test: Increase logging levels

Attempt to debug CI failures.

```
FAIL __device-tests__/gutenberg-editor-rendering-media-blocks.test.js (73.496 s)
  ● Gutenberg Editor Rendering Media Blocks test › should be able to render blocks correctly

    unknown method: Not implemented

      at getErrorFromResponseBody (file:/Users/runner/work/gutenberg/gutenberg/node_modules/webdriver/build/utils.js:194:12)
      at NodeJSRequest._request (file:/Users/runner/work/gutenberg/gutenberg/node_modules/webdriver/build/request/index.js:164:23)
      at Browser.wrapCommandFn (file:/Users/runner/work/gutenberg/gutenberg/node_modules/@wdio/utils/build/shim.js:81:29)
```

* refactor: Prefer `action` to `touchAction`

Avoid the deprecated `touchAction` command.

> The `touchAction` command is deprecated and will be removed in a future version. We recommend to use the action command instead with pointer type touch, e.g.:
>
> await browser.action('pointer', {
>   parameters: { pointerType: 'touch' }
> })

https://webdriver.io/docs/api/element/touchAction/

This may also relate to the fact that Mobile JSON Wire Protocol is
deprecated as well.

> Mobile JSON Wire Protocol
>
> The Mobile JSON Wire Protocol is a super set of mobile commands on top of the JSON Wire Protocol. Given this one is deprecated the Mobile JSON Wire Protocol also got deprecated. Appium might still support some of its commands but it is not recommended to use them.

https://webdriver.io/docs/api/protocols#mobile-json-wire-protocol

* Revert "test: Increase logging levels"

This reverts commit 7bb46de.

* test: Set default `swipeFromTo` delay value

This prevents `pause` from being called without a parameter, which
throws an error.

* test: Replace swipe gesture with simple tap (#55427)

It was unclear as to why the swipe gesture failed to close the bottom
sheet, but a simple tap towards the middle-top of the display achieves
the same result.

* [Mobile] - Appium 2 Migration - Updating more utils (#55420)

* Updates the longPressMiddleOfElement util by using WebdriverIO's action. It adds tapping on the element first before long-pressing due to an issue with iOS not triggering the long-press event. The default waitTime was reduced as well to 1000.

* Update selectTextFromElement to WebdriverIO. It also changes the logic for iOS where it uses the "Select All" option from the context menu instead of using "doubleTap". Timeouts have been reduced as well.

* Refactors selectTextFromElement for Android, the previous implementation would only select the word that was located in the middle of the element, with the new changes it will select all text by long-pressing from the start of the element to the end. Unfortunately we can't access the context menu as we do on iOS so it has to be done manually.

* Add inline duration value and adds inline comments to explain the functionality

* [Mobile] - Update closePicker util and updates Image Block test data (#55484)

* Update closePicker util name to closeMediaPicker, it also unifies the logic for both platforms by having the same locator and using WebdriverIO's waitForDisplayed.

* Update test data for the image block, by changing the image caption to avoid issues in CI and autocorrect

* [Mobile] Improve Screen recording on Android for Appium 2 (#55443)

* Improve Screenrecording on Android by making sure of the following:

The sdcard folder should exist in the emulator, if not there's a script that will create it.

For skipped tests, it will remove empty folders that weren't needed.

When having multiple Android emulators opened, it will use the deviceID that Appium is currently using.

* Rename getAndroidDeviceID function to getDeviceID since it's not platform specific

* Fix comment

* Temp: log session

* Use adb devices list to get the first active android emulator

* Moves getAndroidEmulatorID to its own file and replaces its usage. It also adds specifying the device ID for Android local builds for cases when multiple devices/emulators are connected.

* Remove unneeded filtering

* Move setting the androidDeviceID value for Android cases only

* docs: Update E2E test documentation (#55519)

* docs: Rewrite end-to-end testing documentation

Update for breaking changes from upgrading Appium 2. Reduce verbosity
and duplication to improve the approachability and maintainability of
the documentation.

* docs: Remove E2E test contributing documentation

This was replaced by a section in the directory's README documentation.

* docs: Rewrite end-to-end testing Getting Started section

Update for breaking changes from upgrading Appium 2. Reduce verbosity
and duplication between the `WordPress/gutenberg` repository to improve
the approachability and maintainability of the documentation.

* docs: Update links in Gutenberg for Mobile Apps code documentation

Documents have been updated and consolidated to reduce verbosity
and duplication to improve the approachability and maintainability of
the documentation.

* [Mobile] Remove `wd` usage and removes disabled tests (#55541)

* Device tests - Removes skipped tests that haven't been enabled back for a while. These were disabled due to flakiness most likely to the interactions with the context menu to copy/paste text. If needed this can be restored once the flakiness issues are worked on.

* Device Tests - Drag & Drop: removes disabled tests. These used utils that were flaky and have been disabled for a long time, they can be added back once stable utils are in place for pasting content.

* Editor Page - Removes setHtmlContent since this util is not being used anymore. It can be added back if its needed.

* Device Tests Utils - Removes usage of the wd library. It also removes, doubleTap, clickElementOutsideOfTextInput, and related utils to the context menu due to them being flaky and not currenlty in use: tapSelectAllAboveElement, tapCopyAboveElement and tapPasteAboveElement.

* test: Improve E2E test consistency (#55554)

* test: Mitigate conflicts between development server caches

Occasionally, cryptic npm package dependency errors can occur between
test runs. They appear to originate from stale development caches, e.g.
Metro server. Clearing these caches, which E2E tests should not rely
upon mitigates the likelihood of these errors.

* test: Prefix informative E2E test setup logs

Merely an attempt to add structure to the outputted logs.

* test: Avoid setup scripts during test runs

There may be clarity and efficiency by having an explicit setup script.
It was avoid running unnecessary scripts during test runs. It is also
referenced in the current documentation as a script to run
intentionally.

* build: Ignore generated Android files (#55569)

These assets are generated at build time for E2E testing. Seemingly, we
do not leverage these generated files for the binary/bundle that we
provide to the host apps. Therefore, we can safely ignore these files.

* docs: Avoid GMA project name until it is approved

Use of the "Gutenberg for Mobile Apps" name is still under review. In
the meantime, we should use the existing "Mobile Gutenberg" project
name.

---------

Co-authored-by: David Calhoun <github@davidcalhoun.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Mobile App - i.e. Android or iOS Native mobile impl of the block editor. (Note: used in scripts, ping mobile folks to change) [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants